Build a high-level ETL pipeline with source as an API service covering all the below-listed conditions using the opensource bigdata stack. a. Data request 1: Request > Response back from the endpoint (100ms) (every 15m) b. Data request 2: Request > Response back from the endpoint, with pagination, up to 1000 pages (every 1hr) c. Data request 3: Request > Response, -> OK -> wait 5-15 minutes for server to generate response/report, then query again to download (every 90m) d. Data Push: Request > Response, wait 5-15 minutes for the server to generate response/report, then query again to download, then check if successful/failed (every 15m) do this for 500 simultaneous clients

Database System Concepts
7th Edition
ISBN:9780078022159
Author:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Publisher:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Chapter1: Introduction
Section: Chapter Questions
Problem 1PE
icon
Related questions
Question

Build a high-level ETL pipeline with source as an API service covering all the below-listed conditions using the opensource bigdata stack.

a. Data request 1: Request > Response back from the endpoint (100ms) (every 15m)

b. Data request 2: Request > Response back from the endpoint, with pagination, up to 1000 pages (every 1hr)

c. Data request 3: Request > Response, -> OK -> wait 5-15 minutes for server to generate response/report, then query again to download (every 90m)

d. Data Push: Request > Response, wait 5-15 minutes for the server to generate response/report, then query again to download, then check if successful/failed (every 15m)

do this for 500 simultaneous clients 

Expert Solution
trending now

Trending now

This is a popular solution!

steps

Step by step

Solved in 2 steps

Blurred answer
Knowledge Booster
Properties of Different Architectures
Learn more about
Need a deep-dive on the concept behind this application? Look no further. Learn more about this topic, computer-science and related others by exploring similar questions and additional content below.
Similar questions
  • SEE MORE QUESTIONS
Recommended textbooks for you
Database System Concepts
Database System Concepts
Computer Science
ISBN:
9780078022159
Author:
Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Publisher:
McGraw-Hill Education
Starting Out with Python (4th Edition)
Starting Out with Python (4th Edition)
Computer Science
ISBN:
9780134444321
Author:
Tony Gaddis
Publisher:
PEARSON
Digital Fundamentals (11th Edition)
Digital Fundamentals (11th Edition)
Computer Science
ISBN:
9780132737968
Author:
Thomas L. Floyd
Publisher:
PEARSON
C How to Program (8th Edition)
C How to Program (8th Edition)
Computer Science
ISBN:
9780133976892
Author:
Paul J. Deitel, Harvey Deitel
Publisher:
PEARSON
Database Systems: Design, Implementation, & Manag…
Database Systems: Design, Implementation, & Manag…
Computer Science
ISBN:
9781337627900
Author:
Carlos Coronel, Steven Morris
Publisher:
Cengage Learning
Programmable Logic Controllers
Programmable Logic Controllers
Computer Science
ISBN:
9780073373843
Author:
Frank D. Petruzella
Publisher:
McGraw-Hill Education