Welcome to Cogdata’s documentation!

Cogdata is an very efficient manager for large amount of data.

The two main features are data multi-process processing and data grouping management. With very little additional code, you can use cogdata to manage and run custom tasks

Contents:

• Quick Start
  • Installation
  • Initialization
  • Create Dataset
  • Create Task
  • Check Datasets and Tasks
  • Process
  • Merge
  • Split
  • Clean
• cogdata
  • Subpackages
    • cogdata.data_savers package
      • cogdata.data_savers.base_saver module
      • cogdata.data_savers.binary_saver module
      • cogdata.data_savers.tar_saver module
    • cogdata.datasets package
      • cogdata.datasets.binary_dataset module
      • cogdata.datasets.rar_dataset module
      • cogdata.datasets.tar_dataset module
      • cogdata.datasets.zip_dataset module
    • cogdata.tasks package
      • cogdata.tasks.image_text_tokenization_task module
    • cogdata.utils package
      • Subpackages
        • cogdata.utils.cogview package
          • cogdata.utils.cogview.api module
          • cogdata.utils.cogview.sp_tokenizer module
          • cogdata.utils.cogview.unified_tokenizer module
          • cogdata.utils.cogview.vqvae_tokenizer module
          • cogdata.utils.cogview.vqvae_zc module
      • cogdata.utils.eprogress module
      • cogdata.utils.helpers module
      • cogdata.utils.logger module
      • cogdata.utils.progress_record module
      • cogdata.utils.register module
  • cogdata.arguments module
  • cogdata.cli module
  • cogdata.data_manager module
  • cogdata.data_processor module
  • cogdata.process_single_entry module