Tech | Digital Archive Systems Tech Blog

Highlighting TeX with Ace.js

Overview I had an opportunity to highlight TeX with Ace.js, so here are my notes. I referenced the following article. https://banatech.net/blog/view/11 I hope this serves as a helpful reference. Screen Example Demo Site https://nakamura196.github.io/ace_latex/ Repository https://github.com/nakamura196/ace_latex Source Code <html lang="en"> <head> <title>ACE in Action</title> </head> <body> <div id="editor" style="width: 100%; height: 400px; border: 1px solid gray;"></div> <script src="https://cdnjs.cloudflare.com/ajax/libs/ace/1.23.2/ace.js" integrity="sha512-oVyp48/610D5Jo577cvp2vX+Fc0kYaI6s2tZRqBRzjZh7+y/vOEHTzUefbXk/B8P0B76bOK3tL1zeF/QcXlyiA==" crossorigin="anonymous" referrerpolicy="no-referrer" ></script> <script> const text = `\\usepackage{hiragino,ryumin,cid} \\usepackage[dvips]{graphics} \\usepackage{multicol} \\begin{document} テキスト \\右注{（サンプル）}テキスト \\end{document}`; const editor = ace.edit("editor"); // editor.setTheme("ace/theme/monokai"); editor.setFontSize(14); editor.getSession().setMode("ace/mode/latex"); editor.getSession().setUseWrapMode(true); editor.getSession().setTabSize(4); editor.$blockScrolling = Infinity; editor.setOptions({ enableBasicAutocompletion: true, enableSnippets: true, }); editor.setValue(text, 1); </script> </body> </html>

Building a Django CI/CD Environment from GitHub to EC2 Using GitHub Actions (2023 Edition)

Overview I had the opportunity to build a Django CI/CD environment from GitHub to EC2 using GitHub Actions, and here are my notes. The following article was used as a reference. https://qiita.com/fffukken/items/27b0bfa712940914d3f6 I made some updates to the GitHub Actions configuration compared to the above article. GitHub Actions Configuration name: Test and Deploy on: push: branches: [ develop, main ] pull_request: branches: [ develop ] jobs: build: runs-on: ubuntu-latest strategy: max-parallel: 4 matrix: python-version: [3.9, "3.10"] steps: - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Install Dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Run Tests run: | python manage.py makemigrations python manage.py migrate python manage.py test - name: deploy run: | echo "$SECRET_KEY" > secret_key chmod 600 secret_key ssh -oStrictHostKeyChecking=no ${EC2_USER}@${EC2_HOST} -i secret_key "source <仮想環境名>/bin/activate \ && cd ~/<プロジェクト名>\ && git pull origin main \ && python manage.py makemigrations \ && python manage.py migrate \ && deactivate \ && sudo systemctl restart gunicorn" env: SECRET_KEY: ${{ secrets.SECRET_KEY }} EC2_USER: ${{ secrets.EC2_USER }} EC2_HOST: ${{ secrets.EC2_HOST }} The changes made were updating the versions of actions/checkout and actions/setup-python. I also changed the pip install section to pip install -r requirements.txt. ...

django-simple-history: Recording Model Edit History in Django

Overview This is a personal note from researching how to keep edit history in Django. As shown in the following message, by default, edit history is recorded for changes made through the admin interface, but not for changes made through other interfaces. This object doesn’t have a change history. It probably wasn’t added via this admin site. django-simple-history So I tried using the following package. https://django-simple-history.readthedocs.io/en/latest/ I was able to use it without issues by following the quick start page below. ...

Implementing Partial Match Filters in Django Rest Framework (DRF)

To implement partial match filters in Django Rest Framework (DRF), it is common to use the Django filter backend. This uses the django_filters module. If you haven’t installed this module yet, you can install it with the following command: pip install django-filter Here are the general steps to achieve partial match search: Define a filter set. The following is a filter set for performing partial match search on the name field of a model called MyModel in models.py: ...

Trying Out bagit-python

bagit is described as follows: bagit is a Python library and command line utility for working with BagIt style packages. I created a Google Colab notebook for trying out this library. https://colab.research.google.com/github/nakamura196/ndl_ocr/blob/main/bagit_python.ipynb I hope this serves as a useful reference for using bagit.

Getting the File Path of an Uploaded File Using Django's ModelForm

I had the opportunity to get the file path of an uploaded file using Django’s ModelForm, so I am making a note of it. Assume the following model. class Document(models.Model): file = models.FileField(upload_to='documents/') With the above, the path could be accessed using the following views. from django.shortcuts import render, redirect from .forms import DocumentForm def upload_file(request): if request.method == 'POST': form = DocumentForm(request.POST, request.FILES) if form.is_valid(): document = form.save() file_url = document.file.url # Correct field name used here full_path = document.file.path # Correct field name used here return redirect('some-view') else: form = DocumentForm() return render(request, 'upload.html', {'form': form}) This may be basic, but I hope you find it helpful. ...

Connecting Django with AWS OpenSearch

Overview These are notes on how to connect Django with AWS OpenSearch. The following article was helpful. https://testdriven.io/blog/django-drf-elasticsearch/ However, since the above article targets Elasticsearch, changes corresponding to OpenSearch were needed. Changes Changes for OpenSearch were needed starting from the Elasticsearch Setup section of the article. https://testdriven.io/blog/django-drf-elasticsearch/#elasticsearch-setup Specifically, the following two libraries were required. (env)$ pip install opensearch-py (env)$ pip install django-opensearch-dsl After that, by replacing django_elasticsearch_dsl with django-opensearch-dsl and elasticsearch_dsl with opensearchpy, I was able to proceed as described in the article. ...

Bulk Registration with Django REST Framework

Overview I looked into how to perform bulk registration with Django REST framework, so here are my notes. By following the article below, I was able to create an endpoint for bulk registration. https://qiita.com/Utena-lotus/items/c7bde7f663cfc4aabff1 Postman I sent the following request using Postman. As a result, I was able to perform bulk registration as shown below. Summary I hope this serves as a helpful reference.

Using JWT in Django (djangorestframework-simplejwt)

Overview I wanted to use JWT with Django, so I tried djangorestframework-jwt. https://github.com/jpadilla/django-rest-framework-jwt I followed the instructions on the following site. https://dev-yakuza.posstree.com/django/jwt/ However, when I added 'rest_framework_jwt.authentication.JSONWebTokenAuthentication', the following error occurred. ImportError: cannot import name 'smart_text' from 'django.utils.encoding' Upon investigation, I found the following article. https://stackoverflow.com/questions/72102911/could-not-import-rest-framework-jwt-authentication-jsonwebtokenauthentication It suggested using djangorestframework-simplejwt instead. https://github.com/jazzband/djangorestframework-simplejwt Below are my notes on how to use it. djangorestframework-simplejwt I was able to verify its operation by following the page below. ...

Handling CORS Errors with Drupal's JSON:API

Overview When using the output from Drupal’s JSON:API in a separate application, a CORS error occurred. Here, I explain how to resolve the CORS error. Solution Copy the following file: /web/sites/default/default.services.yml cp /web/sites/default/default.services.yml /web/sites/default/services.yml Then, set enabled to true in cors.config: # Configure Cross-Site HTTP requests (CORS). # Read https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS # for more information about the topic in general. # Note: By default the configuration is disabled. cors.config: enabled: false # Change this to true! # Specify allowed headers, like 'x-allowed-header'. allowedHeaders: [] As a result, the CORS error was resolved. ...

Troubleshooting JSON:API Related Errors

Overview The following JSON:API related error occurred. This is a note on how to resolve it. サイトに予期せぬエラーが起こりました。しばらくたってから再度お試しください。 TypeError: Drupal\jsonapi_search_api_facets\Plugin\facets\facet_source\JsonApiFacets::__construct(): Argument #6 ($index) must be of type Drupal\search_api\IndexInterface, null given, called in /home/j-soken/drupal/web/modules/contrib/jsonapi_search_api/modules/jsonapi_search_api_facets/src/Plugin/facets/facet_source/JsonApiFacets.php on line 61 in Drupal\jsonapi_search_api_facets\Plugin\facets\facet_source\JsonApiFacets->__construct() (line 48 of modules/contrib/jsonapi_search_api/modules/jsonapi_search_api_facets/src/Plugin/facets/facet_source/JsonApiFacets.php). Drupal\jsonapi_search_api_facets\Plugin\facets\facet_source\JsonApiFacets::create() (Line: 21) Drupal\Core\Plugin\Factory\ContainerFactory->createInstance() (Line: 83) Drupal\Component\Plugin\PluginManagerBase->createInstance() (Line: 251) facets_system_breadcrumb_alter() (Line: 545) Drupal\Core\Extension\ModuleHandler->alter() (Line: 94) Drupal\Core\Breadcrumb\BreadcrumbManager->build() (Line: 72) Drupal\system\Plugin\Block\SystemBreadcrumbBlock->build() (Line: 171) Drupal\block\BlockViewBuilder::preRender() call_user_func_array() (Line: 101) ... Solution First, enable error display. Add the following to the configuration file. ...

How to Use Drupal JSON:API (include and Multilingual Support)

Overview This is a personal note on how to use Drupal’s JSON:API. This time, I will cover the use of include for taxonomies and multilingual processing. Data As shown below, the taxonomy “Assistant Professor” has been assigned to the position field. /node/5 Additionally, content multilingualization is enabled, so the English versions of the title and position are also displayed as shown below. /en/node/5 JSON:API Since the above content was created with the content type “faculty”, the data list can be retrieved from the following URL. ...

Content Registration and Multilingual Support Using Drupal Key Auth

Overview In the following article, I performed content registration using Python with Basic authentication. This time, I tried API Key Authentication, referring to the following article. https://designkojo.com/post-drupal-using-jsonapi-vuejs-front-end API Key Authentication The following module was used. https://www.drupal.org/project/key_auth A “Key authentication” tab appeared on the user edit screen, allowing an API key to be generated. When using the API key, the following program can be used. import requests endpoint = 'http://{IP address or domain name}/jsonapi/node/article' key = '{API key}' headers = { 'Accept': 'application/vnd.api+json', 'Content-Type': 'application/vnd.api+json', "api-key": key } payload = { "data": { "type": "node--article", "attributes": { "title": "What's up from Python", "body": { "value": "Be water. My friends.", "format": "plain_text" } } } } r = requests.post(endpoint, headers=headers, json=payload) r.json() Notes on Multilingual Support As a note, it appears that creating translation data is not supported. ...

Trying Wagtail

Overview I tried Wagtail, so here are my notes on issues I encountered. I basically followed the tutorial below: https://docs.wagtail.org/en/v5.0.1/getting_started/tutorial.html Search Feature When I added a page with the Japanese title “My First Article,” the following search did not return any results. http://localhost:8000/admin/pages/search/?q=はじめて On the other hand, the following search did return results. It appeared that partial matching for Japanese is not supported by default. http://localhost:8000/admin/pages/search/?q=はじめての記事 Wagtail API Information about the API is available here: ...

Causes and Solutions When Browse Does Not Work in Archivematica

Overview I encountered a bug where clicking Browse in Archivematica did not allow me to view folders or files. Here I introduce the cause and solution. /transfer/ Symptom /administration/storage/ Error retrieving locations: is the storage server running? Please contact an administrator. Accessing the following returned this JSON: /transfer/locations/ { "message": "Error retrieving source directories", "status": "Failure" } Solution Accessing the following showed that the Storage Service was inaccessible. /administration/general/ ...

Bulk Deleting Amazon ECR Repositories

Overview I had an opportunity to bulk delete Amazon ECR repositories, so here are my notes. Please exercise caution when running these commands. Creating a List of Repositories I referenced the following article. https://qiita.com/fk_2000/items/bffd3b1ad6f3ab109766 Run the following command. aws ecr describe-repositories --output json | jq -re ".repositories[].repositoryName" > repository.list On macOS, if you don’t have the jq command, install it with brew install jq. Deletion Run the following command. The --force flag is used to delete even if images exist. ...

Customizing Views for Custom Models in Django REST Framework JSON:API (DJA)

Overview Let’s customize the views for the model added in the following article. Sort Let’s add ordering_fields. ... class UserInfoViewset(ModelViewSet): ordering_fields = ("user_name", ) # Added here queryset = UserInfo.objects.all() serializer_class = UserInfoSerializer def get_object(self): entry_pk = self.kwargs.get("entry_pk", None) if entry_pk is not None: return Entry.objects.get(id=entry_pk).blog return super().get_object() ... As a result, only user_name became selectable in the “Filters” display. For example, sorting by age returned a validation error. Filter ... class UserInfoViewset(ModelViewSet): queryset = UserInfo.objects.all() serializer_class = UserInfoSerializer ordering_fields = ("user_name", ) # Added from here below # override the default filter backends in order to test QueryParameterValidationFilter # without breaking older usage of non-standard query params like `page_size`. filter_backends = ( QueryParameterValidationFilter, OrderingFilter, DjangoFilterBackend, SearchFilter, ) rels = ( "exact", "iexact", "contains", "icontains", "gt", "gte", "lt", "lte", "in", "regex", "isnull", ) filterset_fields = { "id": ("exact", "in"), "user_name": rels } search_fields = ("user_name", ) ... ... With the above, the following filter became possible. ...

Adding Custom Models to Django REST framework JSON:API (DJA)

Overview In the following article, I confirmed the basic operations of Django REST framework JSON:API (DJA). In this article, I will try adding a custom model to DJA. References I will add a UserInfo model, referencing the following article. https://tech-blog.rakus.co.jp/entry/20220329/python Steps Define the Model Add the following: # ユーザ情報を格納する class UserInfo(BaseModel): user_name = models.CharField(verbose_name='ユーザ名',max_length=32) # ユーザ名 birth_day = models.DateField(verbose_name='生年月日') # 生年月日 age = models.PositiveSmallIntegerField(verbose_name='年齢',null=True,unique=False) # 年齢 created_at = models.DateTimeField(verbose_name='作成日時',auto_now_add=True) Build the Database Execute the following: % django-admin makemigrations --settings=example.settings Migrations for 'example': example/migrations/0013_userinfo.py - Create model UserInfo % django-admin migrate --settings=example.settings Operations to perform: Apply all migrations: auth, contenttypes, example, sessions, sites Running migrations: Applying example.0013_userinfo... OK For reference, the following file is created: ...

Trying Django REST Framework JSON:API (DJA)

Overview I had an opportunity to try Django REST framework JSON:API (DJA), so here are my notes. https://django-rest-framework-json-api.readthedocs.io/en/stable/index.html Installation Launch the example app described on the following page. https://django-rest-framework-json-api.readthedocs.io/en/stable/getting-started.html git clone https://github.com/django-json-api/django-rest-framework-json-api.git cd django-rest-framework-json-api python3 -m venv env source env/bin/activate pip install -Ur requirements.txt django-admin migrate --settings=example.settings django-admin loaddata drf_example --settings=example.settings django-admin runserver --settings=example.settings As a result, the following screens were obtained. http://localhost:8000 for the list of available collections (in a non-JSON:API format!), ...

Creating a REST API Using OpenAPI and AWS CDK (OpenSearch Connection and Custom Domain)

Overview I had the opportunity to create a REST API using OpenAPI and AWS CDK, so this is a memo. The following article was very helpful. https://zenn.dev/taroman_zenn/articles/91879cec40627c The project created this time is published in the following repository. https://github.com/nakamura196/CdkOpenapi Connecting to OpenSearch The implementation is done in the following Lambda. https://github.com/nakamura196/CdkOpenapi/blob/main/lambda/search.ts Environment variables need to be passed to the Lambda, and the following was written in the ts file under lib. ...