robots_txt_processor
A robots.txt processor Dart package, converts from text to an easier to manipulate object.
Importing library
import 'package:robots_txt_processor/robots_txt_processor.dart';
Process a robots.txt string with only one user-agent
final RobotsTxt robotsTxt = processor.process('User-agent: googlebot');
print(robotsTxt.groups[0].userAgent);
Process a robots.txt string with only one sitemap
final RobotsTxt robotsTxt = processor.process('Sitemap: http://example.com/sitemap.xml');
print(robotsTxt.sitemap[0]);
More info
- Robots.txt specifications: https://developers.google.com/search/reference/robots_txt?hl=en