Skip to main content

Blog

Learn About Our Meetup

5000+ Members

MEETUPS

LEARN, CONNECT, SHARE

Join our meetup, learn, connect, share, and get to know your Toronto AI community. 

JOB POSTINGS

INDEED POSTINGS

Browse through the latest deep learning, ai, machine learning postings from Indeed for the GTA.

CONTACT

CONNECT WITH US

Are you looking to sponsor space, be a speaker, or volunteer, feel free to give us a shout.

[Discussion] Measuring run-time of complex machine learning pipelines

I am working on a fairly long pipeline for a complex project on my hands at the moment. There are several modules that form part of the pipeline. I am also under a strict end-to-end run-time SLA, so need to make sure my pipeline runs within N seconds. I’m investigating timing mechanisms for ML projects as a result. I’m working with Python.

All I have found in my brief search for mechanisms is the **timeit** to measure execution time of small code snippets, and the simple time() method in the **time** module. This can obviously work, but doesn’t seem to be a integrated elegant way to do it when there are multiple stages in the pipeline where you need to measure execution time, etc.

Does anyone have opinions on what is good, or any alternatives that people use? Cheers!

submitted by /u/humanager
[link] [comments]