ref: e740b731a9796718546ce6e5144742c6ae3fb86c kart/README.md -rw-r--r-- 4.2 KiB View raw
                                                                                
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
# kart

![logo](https://kart.ne02ptzero.me/cdn/00f77b6c222fb97477d1d2a7c8c1766da773862cb02991444715f894bc1307e3.jpg)

_Art crawler_ - [Live Version](https://kart.ne02ptzero.me)

## Table of Contents

* [About the Project](#about-the-project)
* [Getting Started](#getting-started)
  * [Prerequisites](#prerequisites)
  * [Installation](#installation)
  * [Compilation](#compilation)
* [Usage](#usage)
  * [worker](#worker)
  * [sorter](#sorter)
  * [web](#web)
* [Contributing](#contributing)
* [Bug Report](#bug-report)


## About the project

This collections of binaries helps to crawl art around the internet, mainly
from subreddits. There are stored in S3, where you can then sort them, and
expose them in any way you want.

## Getting Started

### Prerequisites

For the worker, you'll need two libraries, which you can install via pip3:

- `imgurpython`
- `gfycat`

In order to sort the images, two external programs are needed:

- `feh`
- `vlc`

Please refer to your distro package manager on how to install these two.

You will need the following API keys:

- `reddit`
- `imgur`
- `gfycat`
- `S3`

Please refer to the documentation on each of those services on how to get a key.
You are not required to use Amazon's S3, any provider that is S3 compatible
will work too.

### Installation

#### Packages

Archlinux users can install the software through the `kart-git` AUR package

#### Binaries

You can directly download the binaires [here](https://cdn.ne02ptzero.me/kart/)

### Compilation
In order to compile, you will need the following programs:

- `golang`
- `cmake`

Clone the repository:

    git clone https://git.mobley.ne02ptzero.me/~louis/kart

`cd` & create the build directory:

    cd kart && mkdir -p build && cd build && cmake ..

Compile everything:

    make

## Usage
### Configuration
One configuration file is used by all the services:

    reddit:
      id: XXX
      secret: XXX
      user_agent: linux:XXX:0.0.1 (by /u/User)
      username: User
      password: Password
      subreddits:
        - pixelart
        - VoxelArt
    
    imgur:
      id: XXX
      secret: XXX
    
    gfycat:
      id: XXX
      secret: XXX
    
    S3:
      endpoint: s3.fr-par.scw.cloud
      region: fr-par
      id: XXX
      secret: XXX
      bucket: my-bucket
      prefix: non-sorted/
      good_prefix: good/
    
    download:
      directory: /tmp/
      database: /var/db/kart.db.sqlite
    
    scripts:
      directory: /usr/local/sbin/
    
    web:
      port: 8080
      views: /usr/share/kart/views
      static: /usr/share/kart/static
      database: /var/db/kart-web.db.sqlite
      desc: Some description

### Worker

The worker is programs that crawl the subreddits, indefinetely. You can run it
with:

    kart-worker -conf /etc/kart-conf.yml

It will get the posts of subreddits, look for images, gif or videos, and then
upload them in S3, if the images are new.

### Sorter

The sorter is the progam meant to be run by a human in order to sort the images
from the worker.

    kart-sorter -conf /etc/kart-conf.yml

It will launch a 'sort session' where the user is asked to say "Yes or No" to
each images the worker has crawled. If the user wants them, the content is
moved to another prefix in the S3 bucket, in order to be consumed.

### Web

The web binary is the web server that I use to display the images. It is
launched with:

    kart-web -conf /etc/kart-conf.yml

And it is meant to have a reverse proxy in front of it. Here's the one I use:

    server {
        server_name  kart.ne02ptzero.me;

        location / {
            proxy_pass             http://127.0.0.1:8080/;
        }

        location /cdn {
            rewrite                /cdn/(.*) /good/$1  break;
            proxy_pass             https://art.s3.fr-par.scw.cloud/;
        }
    }

## Contributing

If you want to contribute, please send your patches to
[~louis/kart@lists.mobley.ne02ptzero.me](mailto:~louis/kart@lists.mobley.ne02ptzero.me)
You can look at [git-send-email.io](https://git-send-email.io/) if you need to
setup your mail client.

## Bug Report

Please report your bugs at
[todo/~louis/kart](https://todo.mobley.ne02ptzero.me/~louis/kart) You can also
send an email to
[~louis/kart@todo.mobley.ne02ptzero.me](mailto:~louis/kart@todo.mobley.ne02ptzero.me)