LEX.TXT Essay 004
Essay 004
What Happens When Harvey’s Horizon Scan and India’s Own Legal Tech Tools Meet a Regulator That Bans Scraping?
When the content is law, the State’s refusal should need a reason, and the State should offer a way in.
Cite as Manraj Singh Chandpuri, 'What Happens When Harvey’s Horizon Scan and India’s Own Legal Tech Tools Meet a Regulator That Bans Scraping?' (LEX.TXT, 23 October 2026) <https://manrajchandpuri.com/lex/essays/harvey-meets-a-regulator-that-bans-scraping>.
In this essay
Abstract
Horizon scanners ask a regulator’s website for its pages, repeatedly and through software, and the Department of Telecommunications has withheld the permission that section 43 of the Information Technology Act, 2000 makes decisive for that collection. I argue that this refusal exposes the Indian legal technology industry more than its foreign rivals, and that the State’s refusal over its own law should be tested differently from a private publisher’s, so that when the content is law, the refusal needs a reason and the State offers a way in.
In September 2026, Harvey, a legal artificial intelligence (“AI”) company, described a feature it calls Horizon Scanning. A lawyer describes what she wants to follow, such as “a topic, a region, a named regulator”, and Harvey builds a scan “drawing on more than 12,000 sources across 100+ jurisdictions alongside public sources the user describes”.
Indian builders make similar tools, for example TeamLease RegTech has written that it uses “AI, ML, OCR to crawl over 2000 regulatory websites daily”, and many law firms build bespoke trackers in house to feed their client alerts. Each of these horizon scanners asks a regulator’s website for its pages, repeatedly and through software, and tells a person when something changes.
In my IJLT article on ANI Media Pvt. Ltd. v. OpenAI OpCo LLC, I separated the ‘use’ question of whether the usage of the material is lawful from the ‘access’ question of whether the material was lawfully collected. Section 52(1)(q) of the Copyright Act, 1957 already allows anyone to reproduce matter published in the Official Gazette, so the harder question for a scanner is collection. Section 43 of the Information Technology Act, 2000 (“IT Act”) makes collection actionable when it happens “without permission”, and the Department of Telecommunications (“DoT”) has withheld that permission. I argue that this refusal exposes the Indian legal technology industry more than its foreign rivals, and that the State’s refusal over its own law should be tested differently from a private publisher’s.
How an American regulator speaks to software
The United States Securities and Exchange Commission (“SEC”) states that it “does not allow botnets or automated tools to crawl the site”. It then permits collection at up to “10 requests/second” by software that declares its “user agent”, which is a label telling the server who is asking. The Federal Communications Commission, DoT’s nearest counterpart, offers an application programming interface (“API”), which is a door built for software. Both regulators tell software how to collect rather than forbidding it.
In motion · step through
How a horizon scan asks a regulator for its pages
Follow LEX.SCAN, a hypothetical horizon scanner, as it turns a lawyer’s sentence into automated requests, and compare the doors that two regulators offer to software.
Step 1 of 5 · Describe
A lawyer describes what she wants to follow
Harvey’s Horizon Scanning begins with a description, such as a topic, a region or a named regulator. Here a lawyer asks LEX.SCAN to track right of way orders affecting optical fibre roll-out across Indian states.
Step 2 of 5 · Sources
The scan chooses three sources
The scan settles on DoT’s main website, the Telecom eServices Portal and the Gazette. Nothing has been read yet, so each source waits unchecked, and the scan will check all three every hour.
Step 3 of 5 · Request
Software asks for each page and says who is asking
Every check is an automated request. A careful scanner first asks each site for its
robots.txtfile, and it sends a user agent, the label that tells the server who is asking, with an address at which it can be reached.Step 4 of 5 · Washington
An American regulator tells software how to collect
The SEC does not allow botnets or automated tools to crawl its site, but it permits collection at up to 10 requests per second by software that declares its user agent. A labelled request passes the turnstile, and an unlabelled one is turned back.
The Federal Communications Commission goes further and offers an API, a door built for software.
Step 5 of 5 · Two doors
One department gives two answers
DoT’s main website welcomes every crawler with “Allow: /” in its robots.txt file. The eServices Portal, through which new authorisations under the Telecommunications Act, 2023 must be sought, is closed by its own User Agreement, so the scan marks it as blocked by source terms.
LEX.SCAN is a hypothetical horizon scanner drawn for this essay, and alerts@firm.example is a placeholder address.
Understanding the DoT User Agreement
The Telecom eServices Portal is DoT’s online counter, through which new authorisations under the Telecommunications Act, 2023 must be sought from 25 June 2026. A telecom compliance team or a TMT team or a team building solutions based on DoT updates, would want exactly this listing watched. I reproduce the portal’s User Agreement verbatim below, numbering each paragraph in square brackets as a clause.
“[1] This User Agreement shall apply to all the Users of Telecom eServices Portal.
Definitions
[2] Telecom eServices Portal: The term Telecom eServices Portal includes https://eservices.dot.gov.in/ along with all its sub-domains, and internal and external services serviced by this domain.
[3] Services: The term Services include all authorised services and/or facilities and/or features as are available on the Telecom eServices Portal
[4] Unauthorised activities: The term Unauthorised activities includes any activity which is punishable under section 43 (for eg web scraping, altering source code, hacking, introducing viruses etc.)and Section 45 of the Information Technology Act, 2000 (hereinafter IT Act) or which have been defined as offence under Chapter XI of The IT Act and/or any other activity which is prohibited under any Act, rules, regulations having force of law in India and/or any activity which contravenes access controls/Service use limits set by DoT on the Telecom eServices Portal and/or any security feature developed by DoT on the Telecom eServices Portal and/or any activity which is contrary to any other policies of the Telecom eServices Portal.
[5] User: The term User includes any persons/company/organization/entities/body/association/group of persons/ Authorised Service Provider and/or their affiliates accessing the Telecom eServices Portal and/or using/accessing any Services provided by the Telecom eServices Portal and/or conducting any activity authorised in writing by Department of Telecommunications (hereinafter DoT).
Permitted use
[6] Telecom eServices Portal shall not in any manner be used for any unauthorized activity. The Telecom eServices Portal shall be used only for purposes as authorised by the Department of Telecommunications.
[7] Confidential Information and personally identifiable information of any licensee shall not be disclosed. For any unauthorised disclosure which is not attributable to DoT, DoT shall not be held responsible.
[8] While using the Telecom eServices Portal the user shall comply with all applicable laws, and provide up to date, and correct and accurate information as may be required. The Services provided on the Telecom eServices Portal shall be accessed only through the interfaces expressly authorised by DoT or on the Telecom eServices Portal.
[9] Without limitation of the above, the Users of Telecom eServices Portal shall not:-
(i) Imply or state any affiliation with or endorsement of DoT / without the direct and express consent of such organisation (e.g., representing oneself as an accredited DoT associate).
(ii) Manipulating identifiers in order to disguise the origin of any message or post transmitted through the Telecom eServices Portal
(iii) Monitoring the Telecom eServices Portal availability, performance, or functionality for any competitive purposes.
(iv) Overlaying or otherwise modifying the Telecom eServices Portal or their appearance.
(v) Removing or covering or obscuring any advertisement included on the Telecom eServices Portal
(vi) Renting, leasing, loaning, trading, selling/re-selling access to the Telecom eServices Portal or related data of Telecom eServices Portal
(vii) Selling, sponsoring, or otherwise monetizing any Telecom eServices Portal Service or feature without DoT’s direct and express consent.
(viii) Soliciting or collecting email addresses or other personal information of Telecom eServices Portal users or DoT users in any manner.
(ix) Using, disclosing or distributing any data obtained in violation of this policy.[10] This User Agreement shall be governed by and construed in accordance with the Indian Laws. Any contravention of this User Agreement shall be viewed strictly and appropriate Criminal/ Civil action will be initiated in accordance with the applicable laws.
[11] DoT exclusively reserves the rights, to change or modify or alter this User Agreement(whether completely or partially), from time to time, at its discretion, without any objections, interruptions, grounds, claims or liabilities of any third party/users.
[12] If you do not agree to terms of this User Agreement you are requested not to use the Telecom eServices Portal By accessing Telecom eServices Portal it is deemed that you have given your consent to this User Agreement and shall be bound by the terms of this User Agreement.
Last updated on: 07-08-2025”
What the agreement does to a horizon scanner
Clause 4 names web scraping as unauthorised and adds anything which “contravenes access controls/Service use limits set by DoT”, without stating what those limits are. Clause 8 then requires access “only through the interfaces expressly authorised by DoT”. I found no authorised API or feed, so the only authorised interface appears to be the browser that a scanner replaces.
Clauses 9(vi) and 9(vii) forbid reselling “related data” and “monetizing” any portal “feature”, which describes a company that sells alerts. Clause 9(ix) reaches furthest, because it forbids “Using, disclosing or distributing any data obtained in violation of this policy”, and therefore catches the alert itself.
Clause 12 treats a visit as consent, and Clause 11 lets DoT change the terms “at its discretion”. Builders should therefore keep dated copies, as I proposed in A Wayback Machine for Content-Use Signals. Clause 2 confines the agreement to eservices.dot.gov.in. The robots.txt file of DoT’s main website, which tells crawlers where they may go, meanwhile welcomes every crawler with “Allow: /”.
The drafting appears borrowed, since the GST portal adopted an agreement with the same structure in 2018, and Clause 9(v) forbids removing “any advertisement” from a portal on which I saw none. In the terms of ESSAY 001, DoT is plainly the speaker, but its words appear to have been written for other portals.
In motion · step through
What the User Agreement does to a horizon scanner
Open the eServices Portal’s User Agreement inside LEX.SCAN, and follow the clauses that reach each stage of a scan, from collection to the alert it sends.
Step 1 of 6 · Clause 4
Clause 4 names web scraping as unauthorised
Clause 4 counts web scraping among the activities it treats as punishable under section 43 of the IT Act, and adds any activity that “contravenes access controls/Service use limits set by DoT”. It does not state what those limits are.
Step 2 of 6 · Clause 8
Clause 8 leaves only the browser
Clause 8 requires access “only through the interfaces expressly authorised by DoT”. I found no authorised API or feed, so the only authorised interface appears to be the browser that a scanner replaces, a door built for a person.
Step 3 of 6 · Clause 9
Clause 9 reaches the alert itself
Clauses 9(vi) and 9(vii) forbid reselling “related data” and “monetizing” any portal “feature”, which describes a company that sells alerts. Clause 9(ix) forbids “Using, disclosing or distributing any data obtained in violation of this policy”, and therefore catches the alert before it is sent.
Step 4 of 6 · Clauses 12 and 11
A visit signs, and the terms can change
Clause 12 treats a visit as consent, and Clause 11 lets DoT change the terms “at its discretion”. Builders should therefore keep dated copies of every version, as I proposed in A Wayback Machine for Content-Use Signals.
Step 5 of 6 · Provenance
The drafting appears borrowed
The GST portal adopted an agreement with the same structure in 2018. Both bar the use of data obtained in violation, both define unauthorised activity by reference to the IT Act, and both deem a visit to be consent. Clause 9(v) even forbids removing “any advertisement” from a portal on which I saw none.
Step 6 of 6 · Statute
“Punishable” meets the words of the statute
Clause 4 describes these activities as “punishable” under sections 43 and 45 of the IT Act. Section 43 makes a person “liable to pay damages by way of compensation”, and section 45 reaches contraventions of rules, regulations, directions or orders made under the Act. The User Agreement does not say which of these it is.
Clause text is quoted from the User Agreement reproduced above, last updated on 07-08-2025. LEX.SCAN and its alert are illustrative.
My own test finds no permission
My IJLT article proposed four considerations for section 43(b), namely technical conduct, declared function, communicated scope and collection within scope. A scanner defeats no barrier, but DoT has published “a reasonably discoverable instruction” in Clauses 4 and 8, so my test finds no permission. I designed that test for private publishers, however. It has nothing to say when the person refusing is the State and the material is law.
Why the State’s server should be read differently
Unlike a private publisher, DoT is bound by the Constitution and by statute. In Harla v State of Rajasthan, the Supreme Court held that a law cannot take effect until it is published or made known so that people can learn what it requires. Section 4(4) of the Right to Information Act, 2005 requires information to be “easily accessible, to the extent possible in electronic format”. Article 14 forbids arbitrary action, a charge that a copied blanket refusal will find hard to answer. I therefore propose a fifth consideration:
“Public channel: Where the person in charge of the computer resource is a public authority, and the resource publishes material that those affected are bound to know, did the authority offer an authorised channel for automated collection, and is any restriction proportionate to a stated concern such as security or server load?”
The portal holds licensees’ confidential material, as Clause 7 recognises. The SEC shows, however, that a speed limit and honest identification protect a server without a ban. My proposal concerns only public pages, never areas behind a login, and it creates no right to an API.
What this means for Indian legal technology
The first implication is an asymmetry between foreign and Indian builders. Harvey can license content, as it has with LexisNexis, and it can leave a restrictive Indian portal out of its sources. An Indian product promising complete coverage of Indian regulators cannot license DoT’s circulars, because DoT offers no channel. It must therefore either collect against Clauses 4 and 8 or admit a gap in its coverage.
The second implication concerns the bespoke trackers that law firms build in house. A firm’s tool that fetches DoT’s listing engages Clause 4 at collection, and the client alert that follows may engage Clause 9(ix) as distribution. When a firm instead asks a vendor’s tool to follow DoT, it is unclear whether the vendor or the firm “secures access” under section 43. A firm’s contract with its vendor should therefore say who bears that risk.
The third implication falls on regulated companies themselves. Clause 5 includes any “Authorised Service Provider” among users. Telecom licensees must use the portal and most need its alerts, so they are the users most clearly bound not to automate them. I make no claim about any company’s conduct. I read the portal manually on 22 October 2026, since automating my own audit would breach Clause 4.
Until the law is clarified, Indian builders should read a portal’s terms before adding it as a source, prefer the Gazette and websites that welcome crawlers, and keep dated records of the terms in force, and this is exactly where my work on RightSignal comes into play.
In motion · step through
What my test finds, and the door the State could open
Run the four considerations from my IJLT article on the eServices Portal, add the public channel question I propose, and follow what the answer means for Indian builders.
Step 1 of 5 · My test
My four considerations find no permission
My IJLT article proposed four considerations for section 43(b). A scanner defeats no barrier and can identify itself honestly, but DoT has communicated its scope in Clauses 4 and 8, and collection falls outside it, so my test finds no permission.
Step 2 of 5 · A fifth question
A fifth question for the State’s own server
I designed that test for private publishers. Where the person in charge is a public authority publishing material that those affected are bound to know, I propose asking whether it offered an authorised channel for automated collection, and whether any restriction is proportionate.
The answer would then be conditional access, which is a proposal, not current law.
Step 3 of 5 · Digest
The digest finds no source software may read
A weekly digest can report a new development on the portal, but it counts no source there that software may read, so the development can only be opened in a browser.
Step 4 of 5 · Exposure
The exposure falls unevenly
A foreign tool can license content and leave the portal out of its sources. An Indian product promising complete coverage cannot license DoT’s circulars, so it must collect against Clauses 4 and 8 or admit a gap.
Sharing the digest with a firm’s team, its clients or a vendor may engage Clause 9(ix), and Clause 5 includes any Authorised Service Provider among users, so licensees are the users most clearly bound not to automate the portal.
Step 5 of 5 · A way in
A way in
DoT could publish an access statement modelled on the SEC’s, with a rate limit and a declared user agent, offer a feed of new circulars, and confine Clause 4 to the areas behind a login. The new door is drawn in dashes because DoT does not yet offer it.
The digest, the share dialog and the licensee card are illustrative. The proposed door is drawn in dashes because DoT does not offer it.
Conclusion
Horizon scanning as seen in Harvey will soon become ordinary infrastructure for Indian law firms and compliance teams. DoT, however, answers the question of how software may read the law it publishes with borrowed words, and no authorised door. The burden falls hardest on Indian builders who cannot license around it. DoT could publish an access statement modelled on the SEC’s, offer a feed of new circulars, and confine Clause 4 to the areas behind a login. My IJLT article concluded that public accessibility “should remain evidence of permission, not an automatic licence for every automated use”. For the State’s own server, I would add that when the content is law, the State’s refusal should need a reason, and the State should offer a way in.