cogdata.utils package

cogdata.utils.eprogress module

class cogdata.utils.eprogress.LineProgress(total=100, symbol='#', width=25, title='')

Bases: cogdata.utils.eprogress.ProgressBar

Normal Line progress bars.

update(progress=0, speed=0)
Parameters

  • progress (int) – Current progress

  • speed (float) – Average process speed(samples/s)

class cogdata.utils.eprogress.MultiProgressManager(*args, **kwargs)

Bases: object

clear()

Remove all progress bar

put(key, progress_bar)

Add a progress bar

Parameters

  • key (str) – The key of the new bar

  • progress_bar (ProgressBar) – An instance of a progress bar

skip_upline()

Skip the first data line if True

update(key, progress, speed=0)

Update status of progress bars and repaint

Parameters

  • key (str) – The key of the progress bar that need to update

  • progress (int) – The current progress number

  • speed (float) – Average process speed(samples/s)

update_title(key, title)

Update the title of a progress bar

Parameters

  • key (str) – The key of the progress bar that need to update title

  • title (str) – New title

class cogdata.utils.eprogress.ProgressBar(width=25, title='')

Bases: object

Base module of all types of process bar.

static filter_str(pending_str)

Filter 、 、

property lock
abstract update(progress=0)

cogdata.utils.helpers module

cogdata.utils.helpers.dir_size(path)

Sum all files’ size in a directory

Returns

The total size of all files in the directory.

Return type

int

cogdata.utils.helpers.format_file_size(fileSize)

Translate file size into B,KB,MB,GB

Returns

a human-read size.

Return type

str

cogdata.utils.helpers.get_last_line(filename)

get last line of a file

Parameters

filename (str) – file name

Returns

last line or None for empty file

Return type

str or None

cogdata.utils.helpers.get_registered_cls(name)

cogdata.utils.logger module

cogdata.utils.logger.get_logger()

Returns a logger that is set by set_logger

cogdata.utils.logger.set_logger(target_path, rank='main')

Set a specific logger for the current process

Parameters

  • target_path (str) – The root folder of all log files

  • rank (str) – The local rank of the current process

cogdata.utils.progress_record module

class cogdata.utils.progress_record.ProgressRecord(log_dir, rank)

Bases: object

Record progress infomation during processing

classmethod get_all(log_dir, n)

Get merged content in all log files

Parameters

  • log_dir (str) – Root folder of log files

  • n (int) – Number of processes

Returns

[(current progress, total progress, average speed)]

Return type

[(int,int,int)]

update(x, y, speed=None)

Record new status in log files

Parameters

  • x (int) – Current progress

  • y (int) – Total progress

cogdata.utils.register module

cogdata.utils.register.register(cls)

Registers for Dataset, Task and Saver