periodic-crawler
    Overview
    Issues
    Discussions
    API Reference
    Versions
    Alternatives
    Code
    Playground
    Contributors

periodic-crawler

periodic-crawler

1.0.4  •  Updated 1 years ago  •  by Ivan Ganev  •  MIT License

Periodic Crawler

Build Status npm

I do NOT recommend using it in real projects.

npm i periodic-crawler

Exsampls

const periodicCrawler = require('periodic-crawler');

async () => {
  const res = await periodicCrawler(['www.example.com/'], {
    proxy: {
      useProxy: false
    },
    settings: {
      pauseMin: 1,
      pauseMax: 2
    }
  });
  console.log(res);
};

// in console:
// [
//   {
//     "url": "www.example.com/",
//     "dowDate": "2019-04-01T02:39:55.268Z",
//     "body": "<!doctype html>\n<html>\n<head></head>\n\n<body>\n<div>\n    <h2>Example Domain</h2>\n    <p>This domain is established to be used for illustrative examples in documents. You may use this\n    domain in examples without prior coordination or asking for permission.</p>\n    <p><a href=\"http://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
//   }
// ]

TODO

Ready for the next level?
Join Openbase's founding team to help us build the ultimate open-source app store, work with the latest technologies, and enjoy great culture, impact and autonomy
Openbase helps developers choose among and use millions of open-source packages, so they can build amazing products faster.
FacebookLinkedIn
© 2020 Devstore, Inc.