Web crawling: what is allowed? 4 criteria for collecting data legally

Web crawling can be done legally. In Decision 2021Do1533, the Supreme Court of Korea ruled not guilty on all three charges brought over collecting data from a competitor’s app API server: violation of the Information and Communications Network Act, obstruction of business by damaging computers(컴퓨터등장애업무방해), and copyright infringement. What matters is the target and the method, in the three areas below.

Area	Legal basis	What to check in the design
Access rights	Information and Communications Network Act §48(1)	Protective measures such as login or CAPTCHA, and whether the terms of service prohibit collection
Personal information	Personal Information Protection Act §15(1)	Whether names or contact details are included. If so, establish the legal basis first
Works and databases	Copyright Act §16, §93(1)(2)	Whether the whole is copied and reprocessed, or only the necessary part is used

① Access rights: is the area restricted?

Article 48(1) of the Information and Communications Network Act prohibits “intruding into an information and communications network without legitimate access authority or beyond the permitted access authority.”

The Supreme Court has set out what counts as a restricted area.

Whether a service provider has restricted access authority must be judged carefully by considering all objectively evident circumstances, such as protective measures and terms of service.

— Supreme Court Decision 2021Do1533, rendered May 12, 2022

robots.txt is not a protective measure. The standard itself (RFC 9309) states that it is “not a substitute for valid content security measures.” Ignoring it does not automatically make crawling illegal, and following it does not guarantee legality. Read it and follow it, but do not rely on it alone.

② Personal information: is data about people included?

Article 15(1) of the Personal Information Protection Act lists the specific cases in which personal information may be collected and used (consent, special provisions in law, entering into or performing a contract, and others).

If names, contact details, or addresses are included, separate regulations apply whether or not crawling is involved. Define the purpose and the legal basis before you start.

③ Works and databases: how much is taken?

A database producer has “the right to reproduce, distribute, broadcast, or transmit the whole or a substantial part” of the database (Copyright Act §93(1)).

This is a common misunderstanding. A single item is not a “substantial part,” but collecting small amounts over a long period is not automatically safe. If repeated, systematic collection ends up having the same effect as copying a substantial part, it is treated as infringement (§93(2)).

Share your collection target and method, and we will review and propose a lawful collection design and feasible scope at no charge. If you are considering outsourcing, data collection projects start from KRW 150,000.

→ Pricing by project type and our work · Ask about collection scope

Beyond the three areas

• Do not overload the server. Keep the collection rate close to human use and limit parallel requests. Sending requests in bursts can disrupt a third party’s service.
• Do not hide that it is a bot. Include the purpose and contact details in the User-Agent.

Checking order

1. Read the target site’s terms of service and robots.txt yourself
2. Determine whether the data is personal information or a substantial part of a database
3. If the case is complex, have it reviewed by a legal professional

Step 1 comes first for a reason. Apart from the law, the terms of service may set the scope, especially for seller and merchant systems used after login. This is a contractual question before it is a legal one, so checking it first avoids rebuilding later.

One more point. 2021Do1533 is a criminal case. A criminal acquittal does not mean there is no civil liability.

Summary

Most data collection can be done without problems. This is why the Supreme Court acquitted on all three charges. Do not access restricted areas, do not touch personal information, do not copy someone else’s database wholesale, and do not overload the server.

The question is usually not “can we” but “how far.” Drawing that line first means nothing has to be undone after it is built.

When you assess this yourself, the difficulty is usually not the legal interpretation but what comes after it: when the terms say nothing explicit, when the target is a page behind login, or when it is unclear whether personal information is included. These cases cannot be decided from the statutes alone. They need to be assessed together with the collection design. Request a collection design review, and we will share review results for each target.

---

Share the URL of the site you want to collect from.

Tell us in one line what you want to collect, and we will check the terms, robots.txt, and whether personal information is involved, then advise whether it is possible and how far it can go. You can decide whether to proceed with development after the review.

→ Ask whether collection is possible · See our work

---

References

• Information and Communications Network Act Article 48(1) · Personal Information Protection Act Article 15(1) · Copyright Act Articles 16, 93(1) and 93(2) [verified: National Law Information Center(국가법령정보센터)]
• Supreme Court Decision 2021Do1533, rendered May 12, 2022 [verified: original judgment; acquitted on all three charges]
• RFC 9309 Robots Exclusion Protocol [verified: IETF, 2022]