sysadmin/openfreemap
1
0
Fork 0
mirror of https://github.com/hyperknot/openfreemap.git synced 2026-05-21 14:02:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

work

Browse Source
This commit is contained in:
Zsolt Ero
2023-12-27 05:12:29 +01:00
parent 857578edce
commit 336a9eda6b
14 changed files with 815 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

176
scripts/cloudflare/r2-dir-list/.gitignore vendored Normal file
View File

@@ -0,0 +1,176 @@
# user specific
src/config.ts
wrangler.toml
# Logs
logs
_.log
npm-debug.log_
yarn-debug.log*
yarn-error.log*
lerna-debug.log*
.pnpm-debug.log*
# Diagnostic reports (https://nodejs.org/api/report.html)
report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json
# Runtime data
pids
_.pid
_.seed
\*.pid.lock
# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov
# Coverage directory used by tools like istanbul
coverage
\*.lcov
# nyc test coverage
.nyc_output
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
.grunt
# Bower dependency directory (https://bower.io/)
bower_components
# node-waf configuration
.lock-wscript
# Compiled binary addons (https://nodejs.org/api/addons.html)
build/Release
# Dependency directories
node_modules/
jspm_packages/
# Snowpack dependency directory (https://snowpack.dev/)
web_modules/
# TypeScript cache
\*.tsbuildinfo
# Optional npm cache directory
.npm
# Optional eslint cache
.eslintcache
# Optional stylelint cache
.stylelintcache
# Microbundle cache
.rpt2_cache/
.rts2_cache_cjs/
.rts2_cache_es/
.rts2_cache_umd/
# Optional REPL history
.node_repl_history
# Output of 'npm pack'
\*.tgz
# Yarn Integrity file
.yarn-integrity
# dotenv environment variable files
.env
.env.development.local
.env.test.local
.env.production.local
.env.local
# parcel-bundler cache (https://parceljs.org/)
.cache
.parcel-cache
# Next.js build output
.next
out
# Nuxt.js build / generate output
.nuxt
dist
# Gatsby files
.cache/
# Comment in the public line in if your project uses Gatsby and not Next.js
# https://nextjs.org/blog/next-9-1#public-directory-support
# public
# vuepress build output
.vuepress/dist
# vuepress v2.x temp and cache directory
.temp
.cache
# Docusaurus cache and generated files
.docusaurus
# Serverless directories
.serverless/
# FuseBox cache
.fusebox/
# DynamoDB Local files
.dynamodb/
# TernJS port file
.tern-port
# Stores VSCode versions used for testing VSCode extensions
.vscode-test
# yarn v2
.yarn/cache
.yarn/unplugged
.yarn/build-state.yml
.yarn/install-state.gz
.pnp.\*
# wrangler project
.dev.vars
.wrangler/

21
scripts/cloudflare/r2-dir-list/LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2023 Mingjun Cao
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

40
scripts/cloudflare/r2-dir-list/README.md Normal file
View File

@@ -0,0 +1,40 @@
# R2 Directory Listing
This is a simple Directory Listing script for [Cloudflare R2](https://developers.cloudflare.com/r2/) and hosted on [Cloudflare Workers](https://workers.cloudflare.com/). It is inspired by the [Directory Listing of Gitea downloads site](https://blog.gitea.com/evolution-of-the-gitea-downloads-site/).
## Usage
Clone this repository, install dependencies and edit the configs:
```bash
git clone https://github.com/cmj2002/r2-dir-list.git
cd r2-dir-list
npm install
mv src/config.ts.example src/config.ts
mv wrangler.toml.example wrangler.toml
```
You should edit:
- `bucketname` in `src/config.ts` and `wrangler.toml` to your bucket name.
- `bucketdomain.example.com` in `src/config.ts` and `wrangler.toml` to your bucket domain. **It must have been set as a [custom domain](https://developers.cloudflare.com/r2/buckets/public-buckets/#custom-domains) of your Cloudflare R2 bucket**.
- `example.com` in `wrangler.toml`'s `zone_name` to yours.
- Other settings like `name`, `desp`, `showPoweredBy` and `legalInfo` in `src/config.ts` to your own.
You may want to search `bucketdomain`, `bucketname` and `example.com` in your code to ensure you have edited all of them.
Then you can run `wrangler deploy` to deploy it to your Cloudflare Workers.
## Demo
https://datasets.caomingjun.com/
This is a production website of mine, which hosts some machine learning datasets used in my papers and codes to share with other reserchers. It is a Cloudflare R2 bucket with this worker in front of it.
## How it works
It will only overwrite the response when all of the following conditions are met:
- Response from R2 has a status of 404
- The requested pathname ends with `/`
- There exist "subdirectories" or "files" under the current "directory" (The quotation marks here are used because directories and files are abstract concepts in object storage)
In such a case, it will generate a HTML page with the list of "subdirectories" and "files" under the current "directory" and return it. Otherwise, it will just return the response from R2. So **putting this worker in front of your R2 bucket will not affect any normal access to your bucket**.

15
scripts/cloudflare/r2-dir-list/package.json Normal file
View File

@@ -0,0 +1,15 @@
{
"name": "s3-dir-list",
"version": "0.0.0",
"private": true,
"scripts": {
"deploy": "wrangler deploy",
"dev": "wrangler dev",
"start": "wrangler dev"
},
"devDependencies": {
"@cloudflare/workers-types": "^4.20230419.0",
"typescript": "^5.0.4",
"wrangler": "^3.0.0"
}
}

43
scripts/cloudflare/r2-dir-list/src/index.ts Normal file
View File

@@ -0,0 +1,43 @@
import { Env } from './types';
import { renderTemplFull } from './render';
import { getSiteConfig } from './config';
export default {
async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
const originResponse = await fetch(request);
// if status is not 404 or request path not end with '/', return origin response
if ((originResponse.status !== 404) || (originResponse.url.slice(-1) !== '/')) {
return originResponse;
}
const url = new URL(request.url);
const domain = url.hostname;
const path = url.pathname;
// remove the leading '/'
const objectKey = path.slice(1);
const siteConfig = getSiteConfig(env, domain);
if (!siteConfig) {
// TODO: Should send a email to notify the admin
return originResponse;
}
const bucket = siteConfig.bucket;
const index = await bucket.list({
// TODO: Should manage limit here, but since I won't have more than 1000 files in a single folder, I just ignore it
prefix: objectKey,
delimiter: '/',
include: ['httpMetadata', 'customMetadata']
});
// if no object found, return origin response
if (index.objects.length === 0 && index.delimitedPrefixes.length === 0) {
return originResponse;
}
return new Response(
renderTemplFull(index.objects, index.delimitedPrefixes, path, siteConfig),
{
headers: {
'Content-Type': 'text/html; charset=utf-8',
},
status: 200,
},
);
},
};

197
scripts/cloudflare/r2-dir-list/src/render.ts Normal file
View File

@@ -0,0 +1,197 @@
import { svgs, cssStyle, defaultFavicon } from './static';
import { SiteConfig } from './types';
export var renderTemplFull = (files: R2Object[], folders: string[], path: string, config: SiteConfig) => {
return `<!DOCTYPE html>
<html>
<head>
<link rel="icon" href="${config.favicon??defaultFavicon}" type="image/png">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>${renderTemplTitle(config.name, path)}</title>
${cssStyle}
</head>
<body>
${svgs}
<header>
<h1>
<a href="/">${config.name}</a> /
<!-- breadcrumbs start -->${renderTemplBreadcrumbs(path)}
</h1>
</header>
<main>
<div class="listing">
<table aria-describedby="summary">
<thead>
<tr>
<th class="hideable"></th>
<th class="name">Name</th>
<th class="description">Description</th>
<th class="size">Size</th>
<th class="date hideable">Modified</th>
<th class="hideable"></th>
</tr>
</thead>
<tbody>
${(path==='/')?'':renderGoUp(path)}
<!-- folders start -->${renderTemplFolders(folders, config)}
<!-- files start -->${renderTemplFiles(files, config)}
</tbody>
</table>
</div>
</main>
<footer>
${generateFooter(config, path)}
</footer>
</body>
</html>
`;
}
var renderGoUp = (path: string) => {
if(path !== "") {
return `
<tr>
<td class="hideable"></td>
<td class="goup">
<a href="..">
Go up
</a>
</td>
<td class="description">&mdash;</td>
<td class="size">&mdash;</td>
<td class="date hideable">&mdash;</td>
<td class="hideable"></td>
</tr>`;
}
return '';
}
var renderTemplTitle = (siteTitle:string, path: string) => {
if(path === "/") {
return siteTitle
}
path = path.slice(0, -1);
return `${siteTitle} | ${cleanTitle(path)}`;
}
var cleanTitle = (path: string) => {
var parts = path.split("/")
// remove the empty strings
parts = parts.filter((part) => part !== "")
return parts[parts.length-1]
}
var renderTemplBreadcrumbs = (path: string) => {
const parts = path.split('/');
var output = '';
var currentPath = '/';
for (var i = 0; i < parts.length; i++) {
if (parts[i] === '') continue;
currentPath += parts[i] + '/';
output += `<a href="${currentPath}">${parts[i]}</a> / `;
}
console.log(output);
return output;
}
var renderTemplFolders = (folders: string[], siteConfig: SiteConfig) => {
if(typeof folders === 'undefined') return '';
var output = '';
for (var i = 0; i < folders.length; i++) {
output += `<tr class="file ">
<td class="hideable"></td>
<td class="name"><a href="/${folders[i]}"><svg width="1.5em" height="1em" version="1.1" viewBox="0 0 317 259"><use xlink:href="#folder"></use></svg><span class="name">${cleanFolderName(folders[i])}</span></a></td>
<td class="description">${findDesp(siteConfig, '/'+folders[i].slice(0, -1), true)??"&mdash;"}</td>
<td class="size">&mdash;</td>
<td class="date hideable">&mdash;</td>
<td class="hideable"></td>
</tr>`;
}
return output;
}
var renderTemplFiles = (files: R2Object[], siteConfig: SiteConfig) => {
if(typeof files === 'undefined') return '';
var output = '';
for (var i = 0; i < files.length; i++) {
output += `<tr class="file ">
<td class="hideable"></td>
<td class="name"><a href="/${files[i].key}"><svg width="1.5em" height="1em" version="1.1" viewBox="0 0 265 323"><use xlink:href="#file"></use></svg><span class="name">${cleanFileName(files[i].key)}</span></a></td>
<td class="description">${findDesp(siteConfig, '/'+files[i].key, true)??"&mdash;"}</td>
<td class="size">${humanFileSize(files[i].size)}</td>
<td class="date hideable"><time datetime="${files[i].uploaded.toUTCString()}">${files[i].uploaded.toJSON()}</time></td>
<td class="hideable"></td>
</tr>`;
}
return output;
}
var cleanFileName = (name: string) => {
return name.split("/").slice(-1).pop()!
}
var cleanFolderName = (name: string) => {
return name.slice(0, -1).split("/").slice(-1).pop()!
}
// taken from https://stackoverflow.com/questions/10420352/converting-file-size-in-bytes-to-human-readable-string
var humanFileSize = (bytes: number, si=false, dp=1) => {
const thresh = si ? 1000 : 1024;
if (Math.abs(bytes) < thresh) {
return bytes + ' B';
}
const units = si
? ['kB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB']
: ['KiB', 'MiB', 'GiB', 'TiB', 'PiB', 'EiB', 'ZiB', 'YiB'];
let u = -1;
const r = 10**dp;
do {
bytes /= thresh;
++u;
} while (Math.round(Math.abs(bytes) * r) / r >= thresh && u < units.length - 1);
return bytes.toFixed(dp) + ' ' + units[u];
}
function findDesp(siteConfig: SiteConfig, path: string, exact: boolean): string | undefined {
if (exact) {
return siteConfig.desp[path];
}
const keys = Object.keys(siteConfig.desp);
// find the longest match
let longestMatch = '/';
for (const key of keys) {
if (path.startsWith(key) && key.length > longestMatch.length) {
longestMatch = key;
}
}
const desp = siteConfig.desp[longestMatch];
return desp;
}
function generateFooter(siteConfig: SiteConfig, path: string): string {
/// Footer includes:
/// - desp of current path, use the most specific entry
/// - legal info, if any
/// - reference to gitea download site code, as the inspiration of this project
let contents:string[] = [];
const desp = findDesp(siteConfig, path, false);
if (desp) {
contents.push(`<p>${desp}</p>`);
}
if (siteConfig.legalInfo) {
contents.push(`<p>${siteConfig.legalInfo}</p>`);
}
if (siteConfig.showPoweredBy) {
contents.push(`<p>Powered by <a href="https://github.com/cmj2002/r2-dir-list">r2-dir-list</a></p>`);
}
return contents.join('');
}

206
scripts/cloudflare/r2-dir-list/src/static.ts Normal file
View File

File diff suppressed because one or more lines are too long

14
scripts/cloudflare/r2-dir-list/src/types.ts Normal file
View File

@@ -0,0 +1,14 @@
export type Env = {
[K in `BUCKET_${string}`]: R2Bucket;
};
export interface SiteConfig {
name: string;
bucket: R2Bucket;
desp: {
[path: string]: string;
};
legalInfo?: string;
showPoweredBy?: boolean;
favicon?: string;
}

101
scripts/cloudflare/r2-dir-list/tsconfig.json Normal file
View File

@@ -0,0 +1,101 @@
{
"compilerOptions": {
/* Visit https://aka.ms/tsconfig.json to read more about this file */
/* Projects */
// "incremental": true, /* Enable incremental compilation */
// "composite": true, /* Enable constraints that allow a TypeScript project to be used with project references. */
// "tsBuildInfoFile": "./", /* Specify the folder for .tsbuildinfo incremental compilation files. */
// "disableSourceOfProjectReferenceRedirect": true, /* Disable preferring source files instead of declaration files when referencing composite projects */
// "disableSolutionSearching": true, /* Opt a project out of multi-project reference checking when editing. */
// "disableReferencedProjectLoad": true, /* Reduce the number of projects loaded automatically by TypeScript. */
/* Language and Environment */
"target": "es2021" /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */,
"lib": ["es2021"] /* Specify a set of bundled library declaration files that describe the target runtime environment. */,
"jsx": "react" /* Specify what JSX code is generated. */,
// "experimentalDecorators": true, /* Enable experimental support for TC39 stage 2 draft decorators. */
// "emitDecoratorMetadata": true, /* Emit design-type metadata for decorated declarations in source files. */
// "jsxFactory": "", /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h' */
// "jsxFragmentFactory": "", /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */
// "jsxImportSource": "", /* Specify module specifier used to import the JSX factory functions when using `jsx: react-jsx*`.` */
// "reactNamespace": "", /* Specify the object invoked for `createElement`. This only applies when targeting `react` JSX emit. */
// "noLib": true, /* Disable including any library files, including the default lib.d.ts. */
// "useDefineForClassFields": true, /* Emit ECMAScript-standard-compliant class fields. */
/* Modules */
"module": "es2022" /* Specify what module code is generated. */,
// "rootDir": "./", /* Specify the root folder within your source files. */
"moduleResolution": "node" /* Specify how TypeScript looks up a file from a given module specifier. */,
// "baseUrl": "./", /* Specify the base directory to resolve non-relative module names. */
// "paths": {}, /* Specify a set of entries that re-map imports to additional lookup locations. */
// "rootDirs": [], /* Allow multiple folders to be treated as one when resolving modules. */
// "typeRoots": [], /* Specify multiple folders that act like `./node_modules/@types`. */
"types": ["@cloudflare/workers-types/2023-03-01"] /* Specify type package names to be included without being referenced in a source file. */,
// "allowUmdGlobalAccess": true, /* Allow accessing UMD globals from modules. */
"resolveJsonModule": true /* Enable importing .json files */,
// "noResolve": true, /* Disallow `import`s, `require`s or `<reference>`s from expanding the number of files TypeScript should add to a project. */
/* JavaScript Support */
"allowJs": true /* Allow JavaScript files to be a part of your program. Use the `checkJS` option to get errors from these files. */,
"checkJs": false /* Enable error reporting in type-checked JavaScript files. */,
// "maxNodeModuleJsDepth": 1, /* Specify the maximum folder depth used for checking JavaScript files from `node_modules`. Only applicable with `allowJs`. */
/* Emit */
// "declaration": true, /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
// "declarationMap": true, /* Create sourcemaps for d.ts files. */
// "emitDeclarationOnly": true, /* Only output d.ts files and not JavaScript files. */
// "sourceMap": true, /* Create source map files for emitted JavaScript files. */
// "outFile": "./", /* Specify a file that bundles all outputs into one JavaScript file. If `declaration` is true, also designates a file that bundles all .d.ts output. */
// "outDir": "./", /* Specify an output folder for all emitted files. */
// "removeComments": true, /* Disable emitting comments. */
"noEmit": true /* Disable emitting files from a compilation. */,
// "importHelpers": true, /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
// "importsNotUsedAsValues": "remove", /* Specify emit/checking behavior for imports that are only used for types */
// "downlevelIteration": true, /* Emit more compliant, but verbose and less performant JavaScript for iteration. */
// "sourceRoot": "", /* Specify the root path for debuggers to find the reference source code. */
// "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */
// "inlineSourceMap": true, /* Include sourcemap files inside the emitted JavaScript. */
// "inlineSources": true, /* Include source code in the sourcemaps inside the emitted JavaScript. */
// "emitBOM": true, /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */
// "newLine": "crlf", /* Set the newline character for emitting files. */
// "stripInternal": true, /* Disable emitting declarations that have `@internal` in their JSDoc comments. */
// "noEmitHelpers": true, /* Disable generating custom helper functions like `__extends` in compiled output. */
// "noEmitOnError": true, /* Disable emitting files if any type checking errors are reported. */
// "preserveConstEnums": true, /* Disable erasing `const enum` declarations in generated code. */
// "declarationDir": "./", /* Specify the output directory for generated declaration files. */
// "preserveValueImports": true, /* Preserve unused imported values in the JavaScript output that would otherwise be removed. */
/* Interop Constraints */
"isolatedModules": true /* Ensure that each file can be safely transpiled without relying on other imports. */,
"allowSyntheticDefaultImports": true /* Allow 'import x from y' when a module doesn't have a default export. */,
// "esModuleInterop": true /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables `allowSyntheticDefaultImports` for type compatibility. */,
// "preserveSymlinks": true, /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */
"forceConsistentCasingInFileNames": true /* Ensure that casing is correct in imports. */,
/* Type Checking */
"strict": true /* Enable all strict type-checking options. */,
// "noImplicitAny": true, /* Enable error reporting for expressions and declarations with an implied `any` type.. */
// "strictNullChecks": true, /* When type checking, take into account `null` and `undefined`. */
// "strictFunctionTypes": true, /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */
// "strictBindCallApply": true, /* Check that the arguments for `bind`, `call`, and `apply` methods match the original function. */
// "strictPropertyInitialization": true, /* Check for class properties that are declared but not set in the constructor. */
// "noImplicitThis": true, /* Enable error reporting when `this` is given the type `any`. */
// "useUnknownInCatchVariables": true, /* Type catch clause variables as 'unknown' instead of 'any'. */
// "alwaysStrict": true, /* Ensure 'use strict' is always emitted. */
// "noUnusedLocals": true, /* Enable error reporting when a local variables aren't read. */
// "noUnusedParameters": true, /* Raise an error when a function parameter isn't read */
// "exactOptionalPropertyTypes": true, /* Interpret optional property types as written, rather than adding 'undefined'. */
// "noImplicitReturns": true, /* Enable error reporting for codepaths that do not explicitly return in a function. */
// "noFallthroughCasesInSwitch": true, /* Enable error reporting for fallthrough cases in switch statements. */
// "noUncheckedIndexedAccess": true, /* Include 'undefined' in index signature results */
// "noImplicitOverride": true, /* Ensure overriding members in derived classes are marked with an override modifier. */
// "noPropertyAccessFromIndexSignature": true, /* Enforces using indexed accessors for keys declared using an indexed type */
// "allowUnusedLabels": true, /* Disable error reporting for unused labels. */
// "allowUnreachableCode": true, /* Disable error reporting for unreachable code. */
/* Completeness */
// "skipDefaultLibCheck": true, /* Skip type checking .d.ts files that are included with TypeScript. */
"skipLibCheck": true /* Skip type checking all .d.ts files. */
}
}

0
scripts/cloudflare/indexer/.gitignore → scripts/cloudflare/r2-indexer/.gitignore vendored
View File

4
scripts/cloudflare/indexer/package.json → scripts/cloudflare/r2-indexer/package.json
View File

@@ -1,6 +1,4 @@
{
"name": "indexer",
"version": "0.0.0",
"private": true,
"scripts": {
"deploy": "wrangler deploy",
@@ -10,4 +8,4 @@
"devDependencies": {
"wrangler": "^3.0.0"
}
}
}

0
scripts/cloudflare/indexer/src/index.js → scripts/cloudflare/r2-indexer/src/index.js
View File

2
scripts/cloudflare/indexer/wrangler.toml → scripts/cloudflare/r2-indexer/wrangler.toml
View File

@@ -1,4 +1,4 @@
name = "indexer"
name = "r2-indexer"
main = "src/index.js"
compatibility_date = "2023-12-18"

0
scripts/cloudflare/indexer/yarn.lock → scripts/cloudflare/r2-indexer/yarn.lock
View File