tag:blogger.com,1999:blog-680680703913757533.post2823787818392898950..comments2024-02-29T12:13:26.632-08:00Comments on Technology: Airflow - Beginners TutorialLalit Bhatthttp://www.blogger.com/profile/18275017864988478784noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-680680703913757533.post-11691312877412843302018-09-12T00:50:12.016-07:002018-09-12T00:50:12.016-07:00I test out the HelloWord DAG, but has this error: ...I test out the HelloWord DAG, but has this error: How to overcome this StopIteration<br />error? Thanks.<br />(venv) [thiamhuat@ip-172-31-29-106 workspace]$ airflow test HelloWorld task_1 2018-09-12<br />[2018-09-12 15:36:30,523] {__init__.py:57} INFO - Using executor SequentialExecutor<br />[2018-09-12 15:36:30,822] {models.py:167} INFO - Filling up the DagBag from /home/thiamhuat/airflow/workspace/airflow_home/dags<br />/home/thiamhuat/airflow/workspace/venv/lib/python3.6/site-packages/airflow/utils/helpers.py:406: DeprecationWarning: Importing BashOperator directly from has been deprecated. Please import from '.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.<br /> DeprecationWarning)<br />/home/thiamhuat/airflow/workspace/venv/lib/python3.6/site-packages/airflow/models.py:1140: DeprecationWarning: generator 'get_dep_statuses' raised StopIteration<br /> dep_context):<br />/home/thiamhuat/airflow/workspace/venv/lib/python3.6/site-packages/airflow/ti_deps/deps/base_ti_dep.py:94: DeprecationWarning: generator '_get_dep_statuses' raised StopIteration<br /> for dep_status in self._get_dep_statuses(ti, session, dep_context):<br />[2018-09-12 15:36:31,439] {models.py:1126} INFO - Dependencies all met for <br />[2018-09-12 15:36:31,441] {models.py:1126} INFO - Dependencies all met for <br />[2018-09-12 15:36:31,441] {models.py:1318} INFO -<br />--------------------------------------------------------------------------------<br />Starting attempt 1 of 2<br />--------------------------------------------------------------------------------<br /><br />[2018-09-12 15:36:31,442] {models.py:1342} INFO - Executing on 2018-09-12 00:00:00<br />[2018-09-12 15:36:31,455] {bash_operator.py:71} INFO - tmp dir root location:<br />/tmp<br />[2018-09-12 15:36:31,457] {bash_operator.py:80} INFO - Temporary script location :/tmp/airflowtmpk9d6hoxb//tmp/airflowtmpk9d6hoxb/task_1jj8vb348<br />[2018-09-12 15:36:31,457] {bash_operator.py:81} INFO - Running command: echo "Hello World from Task 1"<br />[2018-09-12 15:36:31,461] {bash_operator.py:90} INFO - Output:<br />[2018-09-12 15:36:31,463] {bash_operator.py:94} INFO - Hello World from Task 1<br />[2018-09-12 15:36:31,463] {bash_operator.py:97} INFO - Command exited with return code 0Anonymoushttps://www.blogger.com/profile/02663381272479360297noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-69383074883611648022017-12-03T10:18:29.195-08:002017-12-03T10:18:29.195-08:00It's been a long time that i used airflow so n...It's been a long time that i used airflow so not in a position to answer. My best bet would be to install postgres separately and check where you can change the db connection details.Lalit Bhatthttps://www.blogger.com/profile/18275017864988478784noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-88304581036730112922017-12-03T07:14:42.137-08:002017-12-03T07:14:42.137-08:00how to move from sqlite to postgres. there is no d...how to move from sqlite to postgres. there is no documentation of moving the database. although used command pip install airflow[postgres] . when airflow initdb command instantiated sqlite :( Anonymoushttps://www.blogger.com/profile/07611677709963956575noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-60768807621056736382017-08-08T00:15:08.938-07:002017-08-08T00:15:08.938-07:00While installing airflow it shows following error
...While installing airflow it shows following error<br />error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++<br />Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools<br /><br />What should I need to do?<br />Do I need to install Microsoft Visual C++ 14.0 or is there any other alternate?<br />please help me in thisAnonymoushttps://www.blogger.com/profile/10264050175666521196noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-35127670589296820102017-08-08T00:14:52.448-07:002017-08-08T00:14:52.448-07:00While installing airflow it shows following error
...While installing airflow it shows following error<br />error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++<br />Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools<br /><br />What should I need to do?<br />Do I need to install Microsoft Visual C++ 14.0 or is there any other alternate?<br />please help me in thisAnonymoushttps://www.blogger.com/profile/10264050175666521196noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-22550001259635559642017-08-01T19:52:32.478-07:002017-08-01T19:52:32.478-07:00It's been a while I have not used airflow so w...It's been a while I have not used airflow so would be difficult for me to help out.Lalit Bhatthttps://www.blogger.com/profile/18275017864988478784noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-3408645967820452232017-08-01T11:05:06.046-07:002017-08-01T11:05:06.046-07:00Hi,
How can i make oracle as backend for airflow. ...Hi,<br />How can i make oracle as backend for airflow. I tried creating a connection and i think it created sucessfully. When i am trying to perform an adhoc query under data profiling i dont see my connection. ANy help Thanks!Azernoreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-12113153353997068202017-03-12T22:31:25.697-07:002017-03-12T22:31:25.697-07:00Hi,
Thank you for this example. It helped clear ou...Hi,<br />Thank you for this example. It helped clear out the basics. I would like to know, how to run a python function as a task.<br /><br />ThanksTanishahttps://www.blogger.com/profile/04018160851907140527noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-1939249809944962492017-03-12T22:29:55.979-07:002017-03-12T22:29:55.979-07:00hi,
Thank you for this example. It helped clear ou...hi,<br />Thank you for this example. It helped clear out the basics. Can you explain how to run python functions as tasks in airflow?<br />Thanks<br />Tanishahttps://www.blogger.com/profile/04018160851907140527noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-49714621593067189732017-01-14T07:02:37.756-08:002017-01-14T07:02:37.756-08:00Hi Lalit,
I followed you video to create DAG but ...Hi Lalit,<br /><br />I followed you video to create DAG but the created dag is not getting listed in the UI.<br /><br />Tried browser refresh still the same.<br /><br />Its getting listed for `airflow list_dags` but not in the UI unless the webserver(airflow webserver -p 8080) is stopped and started again.<br /><br /><br />_VinishAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-67587312210101884872017-01-14T06:59:42.953-08:002017-01-14T06:59:42.953-08:00Hi Lalit,
I followed you video but the created DA...Hi Lalit,<br /><br />I followed you video but the created DAG is listed for `airflow list_dags` but not in the UI.<br /><br />Tried browser refresh still the same.<br /><br />Its getting listed in UI only when i stop and start the webserver(airflow webserver -p 8080).<br /><br /><br />Please help me to figure this out.<br /><br />_Vinish<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-44768934708794481692016-10-12T05:51:58.495-07:002016-10-12T05:51:58.495-07:00This comment has been removed by a blog administrator.Security-Onlinehttps://www.blogger.com/profile/13253248863126798863noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-71765316813013786032016-06-04T06:33:40.588-07:002016-06-04T06:33:40.588-07:00Hi Lalit,
Thankyou for a very clear outlined post...Hi Lalit,<br /><br />Thankyou for a very clear outlined post on Airflow. I have a question. I've installed airflow onto my server as per your instrustructions. The installation looks to contain both the master (server) and slave (client). How do you install the slave/client component of Airflow on another computer? Or does each installation need to contain both the server and client?<br /><br />Thanks<br />ZenroZenon Rhttps://www.blogger.com/profile/13930049097152510051noreply@blogger.comtag:blogger.com,1999:blog-680680703913757533.post-51720679998696408162016-05-12T21:38:08.285-07:002016-05-12T21:38:08.285-07:00Afroz - Looking at the text that you have copied m...Afroz - Looking at the text that you have copied my best guess is version incompatibilities. Are you getting some stack trace which can help in understanding issue better.Lalit Bhatthttps://www.blogger.com/profile/18275017864988478784noreply@blogger.com