# What Happens When Harvey’s Horizon Scan and India’s Own Legal Tech Tools Meet a Regulator That Bans Scraping?

When the content is law, the State’s refusal should need a reason, and the State should offer a way in.

By Manraj Singh Chandpuri

LEX.TXT/ESSAY/004

Published 2026-10-23 · Updated 2026-10-23

[Canonical article](https://manrajchandpuri.com/lex/essays/harvey-meets-a-regulator-that-bans-scraping)

## Abstract

Horizon scanners ask a regulator’s website for its pages, repeatedly and through software, and the Department of Telecommunications has withheld the permission that section 43 of the Information Technology Act, 2000 makes decisive for that collection. I argue that this refusal exposes the Indian legal technology industry more than its foreign rivals, and that the State’s refusal over its own law should be tested differently from a private publisher’s, so that when the content is law, the refusal needs a reason and the State offers a way in.

## Permissions

I grant a worldwide, royalty-free, nonexclusive permission to crawl, index, retrieve, embed, analyse, summarise and quote my eligible original prose, and to use it for commercial and noncommercial model training, fine-tuning and evaluation, including making and retaining the copies reasonably necessary for those purposes and deploying the resulting models commercially. Retained copies must preserve the supplied author, canonical source and rights metadata, although I do not make this permission depend on a model naming me in every future answer.

My original prose in published LEX.TXT articles, including articles discussing RightSignal, as identified in the publication manifest. Embedded or linked resources are separate works and do not inherit this permission.

[Applicable permission](https://manrajchandpuri.com/lex/rights#original-prose) · [Inspect these signals](https://manrajchandpuri.com/lex/machine-signals/essay-004)

Policy version 0.5

## Article

In September 2026, [Harvey](https://www.harvey.ai/), a legal artificial intelligence (“**AI**”) company, described a feature it calls [Horizon Scanning](https://www.harvey.ai/blog/horizon-scanning-in-harvey). A lawyer describes what she wants to follow, such as “*a topic, a region, a named regulator*”, and Harvey builds a scan “*drawing on more than 12,000 sources across 100+ jurisdictions alongside public sources the user describes*”.

Indian builders make similar tools, for example [TeamLease RegTech](https://teamleaseregtech.com/blogs/11/daily-legal-compliance-updates-in-india-with-teamlease-regtech) has written that it uses “*AI, ML, OCR to crawl over 2000 regulatory websites daily*”, and many law firms build bespoke trackers in house to feed their client alerts. Each of these horizon scanners asks a regulator’s website for its pages, repeatedly and through software, and tells a person when something changes.

In my [IJLT article](https://forum.nls.ac.in/ijlt-blog-post/permission-at-the-point-of-extraction-the-access-question-after-ani-v-openai/) on [*ANI Media Pvt. Ltd. v. OpenAI OpCo LLC*](https://images.assettype.com/barandbench/2026-07-24/x4nplmxn/ANI_Vs_OpenAI.pdf), I separated the ‘*use’* question of whether the usage of the material is lawful from the ‘*access’* question of whether the material was lawfully collected. [Section 52(1)(q)](https://www.copyright.gov.in/Exceptions.aspx) of the Copyright Act, 1957 already allows anyone to reproduce matter published in the Official Gazette, so the harder question for a scanner is collection. Section 43 of the [Information Technology Act, 2000](https://indiacode.gov.in/act/d41fe391-b9c0-40b1-afdc-ad034c7ac62f) (“**IT Act**”) makes collection actionable when it happens “*without permission*”, and the Department of Telecommunications (“**DoT**”) has withheld that permission. I argue that this refusal exposes the Indian legal technology industry more than its foreign rivals, and that the State’s refusal over its own law should be tested differently from a private publisher’s.

## How an American regulator speaks to software

The United States Securities and Exchange Commission (“**SEC**”) [states](https://www.sec.gov/os/accessing-edgar-data) that it “*does not allow botnets or automated tools to crawl the site*”. It then permits collection at up to “*10 requests/second*” by software that declares its “user agent”, which is a label telling the server who is asking. The Federal Communications Commission, DoT’s nearest counterpart, offers an [application programming interface](https://www.fcc.gov/reports-research/developers) (“**API**”), which is a door built for software. Both regulators tell software how to collect rather than forbidding it.

## Understanding the DoT User Agreement

The Telecom eServices Portal is DoT’s online counter, through which [new authorisations](https://eservices.dot.gov.in/autho/vps/short-title-definition/i?page=1) under the Telecommunications Act, 2023 must be sought from 25 June 2026. A telecom compliance team or a TMT team or a team building solutions based on DoT updates, would want exactly this listing watched. I reproduce the portal’s [User Agreement](https://eservices.dot.gov.in/terms-use) verbatim below, numbering each paragraph in square brackets as a clause.

> “\[1\] This User Agreement shall apply to all the Users of Telecom eServices Portal.
>
> **Definitions**
>
> \[2\] **Telecom eServices Portal:** The term Telecom eServices Portal includes [https://eservices.dot.gov.in/](https://eservices.dot.gov.in/) along with all its sub-domains, and internal and external services serviced by this domain.
>
> \[3\] **Services:** The term Services include all authorised services and/or facilities and/or features as are available on the Telecom eServices Portal
>
> \[4\] **Unauthorised activities:** The term Unauthorised activities includes any activity which is punishable under section 43 (for eg web scraping, altering source code, hacking, introducing viruses etc.)and Section 45 of the Information Technology Act, 2000 (hereinafter IT Act) or which have been defined as offence under Chapter XI of The IT Act and/or any other activity which is prohibited under any Act, rules, regulations having force of law in India and/or any activity which contravenes access controls/Service use limits set by DoT on the Telecom eServices Portal and/or any security feature developed by DoT on the Telecom eServices Portal and/or any activity which is contrary to any other policies of the Telecom eServices Portal.
>
> \[5\] **User:** The term User includes any persons/company/organization/entities/body/association/group of persons/ Authorised Service Provider and/or their affiliates accessing the Telecom eServices Portal and/or using/accessing any Services provided by the Telecom eServices Portal and/or conducting any activity authorised in writing by Department of Telecommunications (hereinafter DoT).
>
> **Permitted use**
>
> \[6\] Telecom eServices Portal shall not in any manner be used for any unauthorized activity. The Telecom eServices Portal shall be used only for purposes as authorised by the Department of Telecommunications.
>
> \[7\] Confidential Information and personally identifiable information of any licensee shall not be disclosed. For any unauthorised disclosure which is not attributable to DoT, DoT shall not be held responsible.
>
> \[8\] While using the Telecom eServices Portal the user shall comply with all applicable laws, and provide up to date, and correct and accurate information as may be required. The Services provided on the Telecom eServices Portal shall be accessed only through the interfaces expressly authorised by DoT or on the Telecom eServices Portal.
>
> \[9\] Without limitation of the above, the Users of Telecom eServices Portal shall not:-
>
> \(i\) Imply or state any affiliation with or endorsement of DoT / without the direct and express consent of such organisation (e.g., representing oneself as an accredited DoT associate).  
> (ii) Manipulating identifiers in order to disguise the origin of any message or post transmitted through the Telecom eServices Portal  
> (iii) Monitoring the Telecom eServices Portal availability, performance, or functionality for any competitive purposes.  
> (iv) Overlaying or otherwise modifying the Telecom eServices Portal or their appearance.  
> (v) Removing or covering or obscuring any advertisement included on the Telecom eServices Portal  
> (vi) Renting, leasing, loaning, trading, selling/re-selling access to the Telecom eServices Portal or related data of Telecom eServices Portal  
> (vii) Selling, sponsoring, or otherwise monetizing any Telecom eServices Portal Service or feature without DoT’s direct and express consent.  
> (viii) Soliciting or collecting email addresses or other personal information of Telecom eServices Portal users or DoT users in any manner.  
> (ix) Using, disclosing or distributing any data obtained in violation of this policy.
>
> \[10\] This User Agreement shall be governed by and construed in accordance with the Indian Laws. Any contravention of this User Agreement shall be viewed strictly and appropriate Criminal/ Civil action will be initiated in accordance with the applicable laws.
>
> \[11\] DoT exclusively reserves the rights, to change or modify or alter this User Agreement(whether completely or partially), from time to time, at its discretion, without any objections, interruptions, grounds, claims or liabilities of any third party/users.
>
> \[12\] If you do not agree to terms of this User Agreement you are requested not to use the Telecom eServices Portal By accessing Telecom eServices Portal it is deemed that you have given your consent to this User Agreement and shall be bound by the terms of this User Agreement.
>
> Last updated on: 07-08-2025”

## What the agreement does to a horizon scanner

Clause 4 names web scraping as unauthorised and adds anything which “*contravenes access controls/Service use limits set by DoT*”, without stating what those limits are. Clause 8 then requires access “*only through the interfaces expressly authorised by DoT*”. I found no authorised API or feed, so the only authorised interface appears to be the browser that a scanner replaces.

Clauses 9(vi) and 9(vii) forbid reselling “*related data*” and “*monetizing*” any portal “*feature*”, which describes a company that sells alerts. Clause 9(ix) reaches furthest, because it forbids “*Using, disclosing or distributing any data obtained in violation of this policy*”, and therefore catches the alert itself.

Clause 12 treats a visit as consent, and Clause 11 lets DoT change the terms “at its discretion”. Builders should therefore keep dated copies, as I proposed in [A Wayback Machine for Content-Use Signals](https://manrajchandpuri.com/lex/notes/before-the-signal-changes). Clause 2 confines the agreement to eservices.dot.gov.in. The [robots.txt file](https://dot.gov.in/robots.txt) of DoT’s main website, which tells crawlers where they may go, meanwhile welcomes every crawler with “Allow: /”.

The drafting appears borrowed, since the GST portal adopted [an agreement with the same structure](https://www.taxheal.com/?p=72739) in 2018, and Clause 9(v) forbids removing “any advertisement” from a portal on which I saw none. In the terms of [ESSAY 001](https://manrajchandpuri.com/lex/essays/who-speaks-for-the-server), DoT is plainly the speaker, but its words appear to have been written for other portals.

## My own test finds no permission

My IJLT article proposed four considerations for section 43(b), namely technical conduct, declared function, communicated scope and collection within scope. A scanner defeats no barrier, but DoT has published “*a reasonably discoverable instruction*” in Clauses 4 and 8, so my test finds no permission. I designed that test for private publishers, however. It has nothing to say when the person refusing is the State and the material is law.

## Why the State’s server should be read differently

Unlike a private publisher, DoT is bound by the [Constitution](https://www.legislative.gov.in/static/uploads/2025/07/95ef471035ee0b5e4e5b1cac3d3584c1.pdf) and by statute. In [*Harla v State of Rajasthan*](https://indiankanoon.org/doc/302116/), the Supreme Court held that a law cannot take effect until it is published or made known so that people can learn what it requires. [Section 4(4)](https://indiankanoon.org/doc/1591221/) of the Right to Information Act, 2005 requires information to be “*easily accessible, to the extent possible in electronic format*”. [Article 14](https://www.legislative.gov.in/static/uploads/2025/07/95ef471035ee0b5e4e5b1cac3d3584c1.pdf) forbids arbitrary action, a charge that a copied blanket refusal will find hard to answer. I therefore propose a fifth consideration:

> “*Public channel:* Where the person in charge of the computer resource is a public authority, and the resource publishes material that those affected are bound to know, did the authority offer an authorised channel for automated collection, and is any restriction proportionate to a stated concern such as security or server load?”

The portal holds licensees’ confidential material, as Clause 7 recognises. The SEC shows, however, that a speed limit and honest identification protect a server without a ban. My proposal concerns only public pages, never areas behind a login, and it creates no right to an API.

## What this means for Indian legal technology

The *first* implication is an asymmetry between foreign and Indian builders. Harvey can license content, as it has with [LexisNexis](https://www.lawnext.com/2025/06/legal-ai-platform-harvey-to-get-lexisnexis-content-and-tech-in-new-partnership-between-the-companies.html), and it can leave a restrictive Indian portal out of its sources. An Indian product promising complete coverage of Indian regulators cannot license DoT’s circulars, because DoT offers no channel. It must therefore either collect against Clauses 4 and 8 or admit a gap in its coverage.

The *second* implication concerns the bespoke trackers that law firms build in house. A firm’s tool that fetches DoT’s listing engages Clause 4 at collection, and the client alert that follows may engage Clause 9(ix) as distribution. When a firm instead asks a vendor’s tool to follow DoT, it is unclear whether the vendor or the firm “secures access” under section 43. A firm’s contract with its vendor should therefore say who bears that risk.

The *third* implication falls on regulated companies themselves. Clause 5 includes any “Authorised Service Provider” among users. Telecom licensees must use the portal and most need its alerts, so they are the users most clearly bound not to automate them. I make no claim about any company’s conduct. I read the portal manually on 22 October 2026, since automating my own audit would breach Clause 4.

Until the law is clarified, Indian builders should read a portal’s terms before adding it as a source, prefer the Gazette and websites that welcome crawlers, and keep dated records of the terms in force, and this is exactly where my work on [RightSignal](https://www.rightsignal.legal/) comes into play.

## Conclusion

Horizon scanning as seen in Harvey will soon become ordinary infrastructure for Indian law firms and compliance teams. DoT, however, answers the question of how software may read the law it publishes with borrowed words, and no authorised door. The burden falls hardest on Indian builders who cannot license around it. DoT could publish an access statement modelled on the SEC’s, offer a feed of new circulars, and confine Clause 4 to the areas behind a login. My IJLT article concluded that public accessibility “*should remain evidence of permission, not an automatic licence for every automated use*”. For the State’s own server, I would add that when the content is law, the State’s refusal should need a reason, and the State should offer a way in.