Hosted edition robots.txt forbids Google et al from indexing the site

Otto wrote...
Hello!

We were wondering why none of our Helpy knowledgebase articles at https://help.seravo.com/ (Helpy hosted edition) never show up on Google, even when very exact search phrases are used.

We found out that the site https://help.seravo.com/robots.txt disallows the indexing of the site. Can you please remove that rule from robots.txt and allow indexing of the knowledgebase?
Flag for Review
      Scott Miller replied...

      Hey Otto,

      Currently only one bot is blocked that demonstrated abusive crawling and Googlebot retains access.  I have verified that Seravo pages show up in the US index of Google here- https://www.google.com/search?q=Which+Seravo.com+Plan+Should+I+Choose%3F .  Do you have a suggestion on what exactly should be changed?


      Flag for Review
        • Seravo google
        Otto replied...
        You are right, the robots.txt format is correct after all. But there is something fishy going on since our Google ranking is so low and the pages seldom come up on the first page even when direct phrases from page header is used as query term.

        Maybe the following matter:
        - The sitemap is empty: https://help.seravo.com/sitemap.xml
        - HTML attributes for lang is English even on pages that are in Finnish:
        $ curl -iLs https://help.seravo.com/fi | grep -i lang
        Flag for Review
            Scott Miller replied...

            Yes these things could be impacting the results.  When I ran some exact searches against the English Google index your support pages came up at the top (when searching the English titles), so I bet it might have something to do with the lang attribute...

            Flag for Review
                Scott Miller replied...

                I did some research and it appears that Google completely ignores the lang attribute, so that won't help.  But a sitemap might.  In addition, adding a hreflang attribute to the links between locales could help.

                https://www.seroundtable.com/google-ignores-the-html-lang-attribute-22974.html

                https://moz.com/learn/seo/hreflang-tag



                Flag for Review
                    Simo Kemppainen replied...
                    Hello!

                    Are you planning to add the sitemap.xml feature to Helpy in the future? I browsed all the settings and didn't find options for that.
                    Flag for Review
                        Scott Miller replied...
                        It is on the roadmap, but I don't have a date for when it might get worked on.  It would be a good contributor project.
                        Flag for Review